New mode of head-polling using client-side state management only by peterbroadhurst · Pull Request #225 · hyperledger-firefly/evmconnect

peterbroadhurst · 2026-08-31T17:48:23Z

There are cases where a stable WebSocket or sticky-session enabled HTTP load balancer is not available to connect to a bank of RPC nodes for an EVM blockchain.

In these cases relying on the server-side filter management provided by execution clients like Besu/Erigon/Reth/Nethermind etc. via eth_newFilter / eth_getFilterChanges is impossible, as every HTTP request gets routed to a new node and the filter is constantly lost and recreated.

This PR proposes a new mode of operation where no server-side state is required during either catchup or steady-state operation of the block listener. This is supported in both light and full block tracking modes.

The architecture of the new mode of operation is described in this comment, and I've called out line-level comments on other areas where I've made changes:

evmconnect/internal/ethereum/event_stream_getlogs.go

Lines 175 to 220 in f466460

    
           // leadGroupSteadyStateGetLogs is the alternative steady state to leadGroupSteadyState, selected with 
        
           // events.filterPollingMode: client. 
        
           // 
        
           // Instead of establishing a node-side filter, we track our own in-memory poll position and page 
        
           // forwards with stateless eth_getLogs range queries. 
        
           // 
        
           // Detection in this function is decoupled from confirmation (in FFTM). 
        
           // The confirmation manager waits for its own configured number of confirmations after each 
        
           // event's block (immediate for an event that arrives already deep enough), 
        
           // delivers to the application, and waits for the ack. 
        
           // 
        
           // For light mode that is based just on a comparison of event vs. head block numbers. 
        
           // For full mode there is a client-side tracking of the full unstable head and the confirmation 
        
           // list is re-calculated client-side. 
        
           // 
        
           // Details on polling approach: 
        
           // 
        
           //   - FULL: We take a snapshot of the block listener's current view of the unstable head 
        
           //     of the chain before we poll for events. Each poll we check if a new fork is apparent 
        
           //     and re-poll from the position of the new fork in the chain. 
        
           //     While we cannot be certain the chain we poll is the same we had a client-side view of 
        
           //     before the poll, we know we will detect further divergence on subsequent poll cycles 
        
           //     (as long as it occurs within the checkpointBlockGap). 
        
           // 
        
           //   - LIGHT: Full blocks are not available to compare, so instead we re-scan the whole unstable 
        
           //     window (the checkpointBlockGap blocks below the head) on every sweep, de-duplicating what 
        
           //     we already delivered by block hash. 
        
           // 
        
           // Checkpoints come from two places: 
        
           // 
        
           //   - Ack-based: each delivered event carries its own checkpoint, persisted by FFTM as batches 
        
           //     are acknowledged. When events are flowing, the checkpoint moves forwards with delivery. 
        
           // 
        
           //   - Inactivity: when no events are flowing, FFTM periodically polls our high water mark (see 
        
           //     getHWM) recording how far we have scanned, held back checkpointBlockGap from the head. 
        
           //     The LastDetected floor in that response stops an inactivity checkpoint overtaking a 
        
           //     detected-but-unacknowledged event, so anything FFTM had not finished delivering is 
        
           //     re-detected after a crash, at any confirmation count. 
        
           // 
        
           // This means on restart, processing will either: 
        
           // 
        
           //   - Continue from the most recent confirmed acknowledged event, which could be the 
        
           //     confirmation count back from the head (configured to 6 or 10 etc.). 
        
           //   - Continue at least checkpointBlockGap behind wherever the head was before the restart, 
        
           //     possibly significantly longer based on when the inactivity checkpoint was captured. 
        
           func (es *eventStream) leadGroupSteadyStateGetLogs() bool {

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst · 2026-08-31T17:58:41Z

 		}
 		// HWM is the configured fromBlock
-		l.hwmBlock = int64(firstBlock) //nolint:gosec // convert to int64 to match the type of hwmBlock, we should change the type of hwmBlock to uint64
+		l.hwmBlock = blockNumberToInt64(firstBlock)


Given this was getting proliferated, I've condensed to a single place and a single behavior if we ever ended up with the (invalid) case of a block in the >maxint64 range.

peterbroadhurst · 2026-08-31T18:00:06Z

+		var exiting bool
+		if es.c.eventFilterPollingMode == FilterPollingModeClient {
+			exiting = es.leadGroupSteadyStateGetLogs()
+		} else {
+			exiting = es.leadGroupSteadyState()
+		}


Deliberately a hard split here, to protect the existing code path from churn.

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

… client-side-filtering

…connect into client-side-filtering

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst · 2026-09-02T16:44:27Z

+		// Catchup only polls blocks that are outside the re-org unstable window at the head of
+		// the chain (checkpointBlockGap behind the head).
+		// The steady-state loops own delivery of the unstable window.
+		// We stop on the first page where the end lands between catchupThreshold+checkpointBlockGap
+		// (say 550) and the checkpointBlockGap (say 50) before the head to do the switch.
+		pollableHead := blockNumberToInt64(chainHeadBlock) - es.c.checkpointBlockGap
+		if pollableHead < 0 {
+			pollableHead = 0
+		}
+		headGap := pollableHead - fromBlock


This might seem decoupled from the primary feature, but this precision on the right change-over point from catchup to steady-state is more important for client-side filtering (particularly in light mode).

In client-side filtering we just maintain a block number, and page forwards from there. Things behind the earliest block we consume from are completely ignored. So the steady state looks a lot like the catchup mode, but we are extra vigilant to re-detect things in the in the checkpointBlockGap (the unstable part).

So it's really important there's no case where this catchup never goes beyond that checkpointBlockGap, and before there was an edge case where the last page could land in that unstable window.

Note the window only existed in the leadGroupCatchup path.

The listenerCatchupLoop that is used when a new listener is added that's behind the lead group already had the strong protection against joining the lead group in the checkpointBlockGap.

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

New mode of head-polling using client-side state management only

8ac9518

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst force-pushed the client-side-filtering branch from b06d585 to 8ac9518 Compare August 31, 2026 17:48

peterbroadhurst added 2 commits August 31, 2026 13:54

Config spelling

c3cc986

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

Config spelling consistency with fitlerPollingInterval

d22041f

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst commented Aug 31, 2026

View reviewed changes

Support light mode for client-side filtering

4e02600

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst added 4 commits September 1, 2026 14:23

Merge branch 'main' of github.com:hyperledger/firefly-evmconnect into…

eec517b

… client-side-filtering

Merge branch 'ws-only-listener' of github.com:hyperledger/firefly-evm…

ed90f2a

…connect into client-side-filtering

Work through light+client mode

466e122

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

Clean up the catchup-to-head boundary consistently

47c9dab

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst commented Sep 2, 2026

View reviewed changes

peterbroadhurst added 3 commits September 2, 2026 17:28

Further clarify the full vs. light mode behavior

d119282

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

Further refinement and clarity on design

2d912d1

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

Fix up the config description

f466460

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>

peterbroadhurst marked this pull request as ready for review September 3, 2026 12:56

peterbroadhurst requested a review from a team as a code owner September 3, 2026 12:56

peterbroadhurst requested a review from Chengxuan September 3, 2026 14:02

Base automatically changed from ws-only-listener to main September 4, 2026 06:24

Navigation Menu

Uh oh!

New mode of head-polling using client-side state management only - #225

New mode of head-polling using client-side state management only#225
peterbroadhurst wants to merge 11 commits into
mainfrom
client-side-filtering

peterbroadhurst commented Aug 31, 2026 •

edited

Loading

Uh oh!

peterbroadhurst Aug 31, 2026

Uh oh!

peterbroadhurst Aug 31, 2026

Uh oh!

peterbroadhurst Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	// leadGroupSteadyStateGetLogs is the alternative steady state to leadGroupSteadyState, selected with
	// events.filterPollingMode: client.
	//
	// Instead of establishing a node-side filter, we track our own in-memory poll position and page
	// forwards with stateless eth_getLogs range queries.
	//
	// Detection in this function is decoupled from confirmation (in FFTM).
	// The confirmation manager waits for its own configured number of confirmations after each
	// event's block (immediate for an event that arrives already deep enough),
	// delivers to the application, and waits for the ack.
	//
	// For light mode that is based just on a comparison of event vs. head block numbers.
	// For full mode there is a client-side tracking of the full unstable head and the confirmation
	// list is re-calculated client-side.
	//
	// Details on polling approach:
	//
	// - FULL: We take a snapshot of the block listener's current view of the unstable head
	// of the chain before we poll for events. Each poll we check if a new fork is apparent
	// and re-poll from the position of the new fork in the chain.
	// While we cannot be certain the chain we poll is the same we had a client-side view of
	// before the poll, we know we will detect further divergence on subsequent poll cycles
	// (as long as it occurs within the checkpointBlockGap).
	//
	// - LIGHT: Full blocks are not available to compare, so instead we re-scan the whole unstable
	// window (the checkpointBlockGap blocks below the head) on every sweep, de-duplicating what
	// we already delivered by block hash.
	//
	// Checkpoints come from two places:
	//
	// - Ack-based: each delivered event carries its own checkpoint, persisted by FFTM as batches
	// are acknowledged. When events are flowing, the checkpoint moves forwards with delivery.
	//
	// - Inactivity: when no events are flowing, FFTM periodically polls our high water mark (see
	// getHWM) recording how far we have scanned, held back checkpointBlockGap from the head.
	// The LastDetected floor in that response stops an inactivity checkpoint overtaking a
	// detected-but-unacknowledged event, so anything FFTM had not finished delivering is
	// re-detected after a crash, at any confirmation count.
	//
	// This means on restart, processing will either:
	//
	// - Continue from the most recent confirmed acknowledged event, which could be the
	// confirmation count back from the head (configured to 6 or 10 etc.).
	// - Continue at least checkpointBlockGap behind wherever the head was before the restart,
	// possibly significantly longer based on when the inactivity checkpoint was captured.
	func (es *eventStream) leadGroupSteadyStateGetLogs() bool {

Uh oh!

Conversation

peterbroadhurst commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peterbroadhurst Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

peterbroadhurst Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

peterbroadhurst Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

peterbroadhurst commented Aug 31, 2026 •

edited

Loading